feat(PLU-441): add OAuth support to GitHub source connector#759
Conversation
Add oauth_token/refresh_token to GithubAccessConfig alongside the existing access_token (PAT), matching the init-oauth-refresh field contract. get_client prefers oauth_token; refresh is handled upstream, not in the connector (PLU-381). PAT and OAuth are mutually exclusive and PAT configs work unchanged. Classify GitHub API errors: 401/403 -> UserAuthError (reconnect-required), 429/rate-limit -> RateLimitError, 5xx -> ProviderError (fixes >500 off-by-one). Add unit tests and an OAuth integration variant; bump to 1.6.29. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…auth Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # CHANGELOG.md # unstructured_ingest/__version__.py # unstructured_ingest/processes/connectors/github.py
Set metadata.version to each file's git blob SHA (element.sha) instead of the tree-response ETag (element.etag). The blob SHA is a content hash already present in the recursive tree listing (no extra API call) and changes iff a file's content changes, which is what the platform's per-record incremental skip logic compares on. The ETag was shared across every file and bumped on any repo change, so it could not drive per-file skip. Null-guard metadata.date_modified (derived from the tree response's Last-Modified header, which the git-data API may omit) so indexing no longer crashes on None. Add unit tests for version and date_modified handling. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 7 files
Shadow auto-approve: would not auto-approve because issues were found.
Re-trigger cubic
dizon-u10d
left a comment
dizon-u10d
left a comment
There was a problem hiding this comment.
I don’t see any blocking issues here. The OAuth/PAT validation, GitHub error classification, retry handling, tree truncation fallback, blob-SHA versioning, and nested-path download fix all line up with the intended behavior.
I also checked the existing PR review state first; there are no prior reviews or unresolved changes-requested blockers to account for.
Validated locally with:
uv run --extra github --group test pytest test/unit/connectors/test_github.py -quv run --extra github --group lint ruff check unstructured_ingest/processes/connectors/github.py test/unit/connectors/test_github.py test/integration/connectors/test_github.pygit diff --check origin/main...HEAD
There was a problem hiding this comment.
0 issues found across 2 files (changes from recent commits).
Shadow auto-approve: would require human review. Adds OAuth authentication and adjusts operational tradeoffs (retry backoff, version metadata change), affecting authorization policy and output contract. Requires human review.
Re-trigger cubic
3 participants
Summary
Prepares the GitHub source connector for Foundation by adding OAuth auth, blob-SHA-based incremental change detection, and hardening indexing/downloading (API-call economy, tree-truncation handling, a download-path bug fix, consistent error classification, and rate-limit-aware retries). PAT configs continue to work unchanged.
What changed
OAuth support.
GithubAccessConfigacceptsoauth_token/refresh_tokenalongside the existingaccess_token(PAT), matching the field contract the platform'sinit-oauth-refreshcontainer expects.oauth_tokentakes precedence; token refresh is handled upstream (PLU-381), not by the connector. PAT and OAuth are mutually exclusive.Blob SHA as
metadata.version. The indexer emits each file's git blob SHA (a content hash already in the tree listing — no extra call) so the platform's per-record incremental skip logic works correctly. The old tree ETag was shared across every file and couldn't drive per-file skip.metadata.date_modifiedis no longer set (git trees carry no reliable per-file timestamp;versionis the sole skip signal).API-call economy + tree truncation. Fetches the repo/default-branch once per run instead of twice per file (
2N+3→2calls). When GitHub truncates the recursive tree (~100k-entry / 7 MB cap, non-paginable), falls back to a per-directory walk so large repos index completely. Skips empty files, directories, and submodules. The downloader caches the repo object per instance (i.e. per process/pod), so repeated downloads in one worker skip the redundant GET /repos/{owner}/{repo} — safe under pod fan-out since nothing is shared across processes.Download-path fix.
GithubDownloader.runnow creates the parent directory chain before writing — previously every download of a nested (or root) path failed withFileNotFoundError.Error classification + retries. GitHub errors map consistently (auth 401/403 →
UserAuthError, throttle →RateLimitError, 5xx →ProviderError; network resets/timeouts →SourceConnectionNetworkError). Transient throttles / 5xx / network errors are retried with exponential backoff that honors GitHub'sRetry-After(secondary limit) andx-ratelimit-reset(primary limit), capped at 300s. A 403 with an exhausted window is treated as a retriable throttle, not a permanent auth failure. Attempt budget viamax_retries(default 10) on both indexer and downloader configs. The raw-blob fetch now uses a bounded request timeout. Addstenacityto thegithubextra.Testing
test/unit/connectors/test_github.py(61 cases) covering auth config, error classification, blob-SHA versioning, recursive listing + truncation walk, downloader caching/fetch, backoff parsing, and the retry loop. All pass in ~0.2s (retries mocked to not sleep).test/integration/connectors/test_github.pyrefactored to share a validation helper; addedtest_github_source_oauthexercising the OAuth token path (requiresGH_OAUTH_ACCESS_TOKEN). Existing PAT test preserved.ruff checkclean.Notes
1.7.1;tenacityadded to thegithubextra with a matchinguv.lockedit (uv lockcouldn't be run in-sandbox due to the Azure-CLI auth requirement, so the lock was updated by hand to match uv's format — worth a quickuv lockconfirmation in CI).platform-etl-orchestration(etl_job_apiincremental registry,etl-operatorGithubOAuthRefresher).